The Significance of Storage in the 'Cost of Risk' of Digital Preservation

As storage costs drop, storage is becoming the lowest cost in a digital repository – and the biggest risk. We examine current modelling of costs and risks in digital preservation, concentrating on the Total Cost of Risk when using digital storage systems for preserving audiovisual material. We present a managed approach to preservation, and the vital role of storage and show how planning for long-term preservation of data should consider the risks involved in using digital storage technology. Gaps in information necessary for accurate modeling – and planning – are presented. We call for new functionality to support recovery of files with errors, to eliminate the all-or-nothing approach of current IT systems, reduce the impact of failures of digital storage technology and mitigate against loss of digital data. Significance of Storage As storage costs continue to drop by roughly 50% every 18 months, there are two effects: o Storage looks free (but isn’t): the cost of storage devices becomes negligible, but power, space, cooling, management and replaced costs remain significant. o Storage is abundant: much more storage is used The following figure shows how hard drive storage has increased over the last 25 years (Hankwang 2008). The largest available size (for a desktop computer) has increased from 5 MB to one terabyte – a factor of 200 000 (which is about 18 doublings in about 25 years, so very close to doubling every 18 months). The ‘growth of risk’ is of course much larger: a factor of 200 000 in disc size, times the increase in the usage of discs (about 10 000 over the same period; Computer World, 2008). This “growth of storage” also divides into two effects: o the number of storage units (globally, and used by any given institution) increases o the amount of data stored on each unit also increases The increase in storage units (devices) means that statistics on failure rates that were once seen as ‘safe’ are now appreciable risks. An advertised Mean Time Between Failure of 1000 years looks very safe to a person buying a new hard drive (though is will be obsolete in 5 years). Schroeder and Gibson (2007) give results on a survey of major datacentres holding 100 000 discs, and found annual failure rates ranging from one to 13 %, averaging around 3% far higher than an MTBF of 1000 years. This failure rate means that owners of 1000 of those same hard drives will need systems (eg big RAID6 arrays) and processes (eg continual hot-swapping and rebuilding) to ensure these failures are managed.. The increase in storage units results in more and more users being responsible for, or dependent upon, storage systems that have thousands of individual storage devices (hard drives, optical discs, data tapes). The increase in the amount of data stored on each device makes the failure of each device more significant in terms of the volume of data potentially lost. A 3.5” floppy disc with 1.4 megabytes (MB) of data represented a few dozen files. A 650 MB CD could hold 500 times more data: thousands of files, or one hour of audio. A USB-attached terabyte hard drive is 700 000 times bigger than a floppy, and 1400 times bigger than a CD. It could, for example, hold the entire contents of an institution’s audio collection (such as several years’ work by many people, collecting oral history recordings).